Concept Drift Adaptation Methods under the Deep Learning Framework: A Literature Review

With the advent of the fourth industrial revolution, data-driven decision making has also become an integral part of decision making. At the same time, deep learning is one of the core technologies of the fourth industrial revolution that have become vital in decision making. However, in the era of epidemics and big data, the volume of data has increased dramatically while the sources have become progressively more complex, making data distribution highly susceptible to change. These situations can easily lead to concept drift, which directly affects the effectiveness of prediction models. How to cope with such complex situations and make timely and accurate decisions from multiple perspectives is a challenging research issue. To address this challenge, we summarize concept drift adaptation methods under the deep learning framework, which is beneficial to help decision makers make better decisions and analyze the causes of concept drift. First, we provide an overall introduction to concept drift, including the definition, causes, types, and process of concept drift adaptation methods under the deep learning framework. Second, we summarize concept drift adaptation methods in terms of discriminative learning, generative learning, hybrid learning, and others. For each aspect, we elaborate on the update modes, detection modes, and adaptation drift types of concept drift adaptation methods. In addition, we briefly describe the characteristics and application fields of deep learning algorithms using concept drift adaptation methods. Finally, we summarize common datasets and evaluation metrics and present future directions.

5GhNet: an intelligent QoE aware RAT selection framework for 5G-enabled healthcare network.

The COVID-19 outbreak has stimulated the digital transformation of antiquated healthcare system to a smart hospital, enabling the personalised and remote healthcare services. To augment the functionalities of these intelligent healthcare systems, 5G & B5G heterogeneous network has emerged as a robust and reliable solution. But the pivotal challenge for 5G & B5G connectivity solutions is to ensure flexible and agile service orchestration with acknowledged Quality of Experience (QoE). However, the existing radio access technology (RAT) selection strategies are incapacitated in terms of QoE provisioning and Quality of Service (QoS) maintenance. Therefore, an intelligent QoE aware RAT selection architecture based on software-defined wireless networking (SDWN) and edge computing has been proposed for 5G-enabled healthcare network. The proposed model leverages the principles of invalid action masking and multi-agent reinforcement learning to allow faster convergence to QoE optimised RAT selection policy. The analytical evaluation validates that the proposed scheme outperforms the other existing schemes in terms of enhancing personalised user-experience with efficient resource utilisation.

HRL4EC: Hierarchical reinforcement learning for multi-mode epidemic control.

Infectious diseases, such as Black Death, Spanish Flu, and COVID-19, have accompanied human history and threatened public health, resulting in enormous infections and even deaths among citizens. Because of their rapid development and huge impact, laying out interventions becomes one of the most critical paths for policymakers to respond to the epidemic. However, the existing studies mainly focus on epidemic control with a single intervention, which makes the epidemic control effectiveness severely compromised. In view of this, we propose a Hierarchical Reinforcement Learning decision framework for multi-mode Epidemic Control with multiple interventions called HRL4EC. We devise an epidemiological model, referred to as MID-SEIR, to describe multiple interventions' impact on transmission explicitly, and use it as the environment for HRL4EC. Besides, to address the complexity introduced by multiple interventions, this work transforms the multi-mode intervention decision problem into a multi-level control problem, and employs hierarchical reinforcement learning to find the optimal strategies. Finally, extensive experiments are conducted with real and simulated epidemic data to validate the effectiveness of our proposed method. We further analyze the experiment data in-depth, conclude a series of findings on epidemic intervention strategies, and make a visualization accordingly, which can provide heuristic support for policymakers' pandemic response.

iMnet: Intelligent RAT Selection Framework for 5G Enabled IoMT Network.

The COVID-19 outburst has encouraged the adoption of Internet of Medical Things (IoMT) network to empower the antiquated healthcare system and alleviate the health care costs. To realise the functionalities of the IoMT network, 5G heterogeneous networks emerged as an exemplary connectivity solution as it facilitates diversified service provisioning in the service delivery model at more convenient care. However, the crucial challenge for 5G heterogeneous wireless connectivity solution is to facilitate agile differentiated service provisioning. Lately, considerable research endeavour has been noted in this direction but multiservice consideration and battery optimisation have not been addressed. Motivated by the gaps in the existing literature, an intelligent radio access technology selection approach has been proposed to ensure Quality of Service provisioning in a multiservice scenario on the premise of battery optimisation. In particular, the proposed approach leverages the concept of Double Deep Reinforcement Learning to attain an optimal network selection policy. Eventually, the proposed approach corroborated by the rigorous simulations demonstrated a substantial improvement in the overall system utility. Subsequently, the performance evaluation underlines the efficacy of the proposed scheme in terms of convergence and complexity.

EdgeVR360: Edge-Assisted Multiuser-Oriented Intelligent 360-degree Video Delivery Scheme over Wireless Networks

The COVID-19 situation has determined many people all over the world to experience remote work, study and play although most of them were not prepared for such a change in their lifestyle. With the coming of the high demand of virtual interaction, 360-degree Virtual Reality (VR) technologies and applications have established stronger relationships with your peers and friends if it applies. However, higher quality of VR streaming brings users deeper immersive experience which requires greater network bandwidth and latency, and more powerful computation capability for individuals. To address these issues, the proposed intelligent video delivery scheme in this paper takes advantage of the edge-assisted computational power to improve the multi-user oriented watching experience of high quality 360-degree video over wireless networks, which reduces network resource utilization, and also optimizes edge cache hit ratio and user's Field of View (FoV) quality.

Digital Twin for Transportation Big Data: A Reinforcement Learning-Based Network Traffic Prediction Approach

Vehicular Ad-Hoc Networks (VANETs), as the crucial support of Intelligent Transportation Systems (ITS), have received great attention in recent years. With the rapid development of VANETs, various services have generated a great deal of data that can be used for transportation planning and safe driving. Especially, with the advent of Coronavirus Disease 2019 (COVID-19), the transportation system has been impacted, thus novel modes of transportation planning and intelligent applications are necessary. Digital twins can provide powerful support for artificial intelligence applications in Transportation Big Data (TBD). The features of VANETs are varying, which arises the main challenge of digital twins applying in TBD. Network traffic prediction, as part of digital twins, is useful for network management and security in VANETs, such as network planning and anomaly detection. This paper proposes a network traffic prediction algorithm aiming at time-varying traffic flows with a large number of fluctuations. This algorithm combines Deep Q-Learning (DQN) and Generative Adversarial Networks (GAN) for network traffic feature extraction. DQN is leveraged to carry out network traffic prediction, in which GAN is involved to represent Q-network. Meanwhile, the generative network can increase the number of samples to improve the prediction error. We evaluate the performance of our method by implementing it on three real network traffic data sets. Finally, we compare the two state-of-the-art competing methods with our method. IEEE

Reviving the economy while saving lives: a deep reinforcement learning approach for smart POI reopening

With the gradual improvements in COVID-19 metrics and the accelerated immunization progress, countries around the world have began to focus on reviving the economy while continuously strengthening epidemic control. POInt-of-Interest (POI) reopening, as a necessity for restoring human mobilities, has become a crucial step to recouple economic recovery and public health management. In contrast to the lock-down policy, POI reopening demands a dynamic trade-off between epidemic interventions and economic costs. In the urban scenario, there exist three key challenges in developing effective POI reopening strategies as follows. (1) During the POI reopening process, there are multiple urban factors affecting the epidemic transmission, which are difficult to simultaneously incorporate and balance in a single reopening strategy;(2) the effects of POI reopening on both economic recovery and epidemic control are long-term, which are hard to capture by static models;and (3) the dual objectives of minimizing infections and maintaining POIs' visits are conflicting, making it difficult to achieve a flexible and scalable trade-off. To tackle the above challenges, we propose Reopener, a deep reinforcement learning (RL) framework for smart POI reopening. First, we utilize a bipartite graph neural network to automatically encode all urban factors that would affect the epidemic prevention and POI visit restriction. Second, we employ a RL-based deep policy network to enable flexible updates in restrictions on POIs along with the trend of epidemic. Third, we design a novel reward function to guide the RL agent to learn smartly, thus comprehensively trading off infections and visit sustainability of POIs. Extensive experimental results demonstrate that Reopener outperforms all baseline methods with remarkable improvements, by reducing the overall economic cost by at least 6.42%. Reopener can effectively suppress infections and support a phase-based POI reopening process, which provides valuable insights for strategy design in post-COVID-19 economic recovery. © 2022 Owner/Author.

Reinforcement Learning and Graph Neural Networks for Designing Novel Drugs with Optimized Affinity: Application to SARS-CoV-2

Drugs are generally designed for a specific target protein. Recent studies have demonstrated the capability of deep learning-based models to accelerate and cheapen the drug development process. The proposed deep learning models can generate novel molecules with optimized drug-like properties. However, the properties addressed are often limited and may be misleading. This is because they do not reflect the complete information about the binding affinity of the designed drug and the target protein. In this work, we exploit the state-of-The-Art progress made in drug-Target-Affinity (DTA) prediction to assess the binding affinity of drugs generated by a developed molecular generator against the corona-virus main protease (SARS-CoV-2 Mpro). The molecular generator is a recurrent neural network-based model, while the DTA predictor is a graph neural network (GNN), famously known as GraphDTA. We train the molecular generator using reinforcement learning (RL) to optimize the GraphDTA-predicted score. As this is the first benchmark of this kind (to the best of our knowledge), we report our benchmarking results;of 1.79% desirability;with the hope of motivating future improvements in this regard. © 2022 IEEE.

Preparing for the next pandemic: Simulation-based deep reinforcement learning to discover and test multimodal control of systemic inflammation using repurposed immunomodulatory agents.

Background: Preparation to address the critical gap in a future pandemic between non-pharmacological measures and the deployment of new drugs/vaccines requires addressing two factors: 1) finding virus/pathogen-agnostic pathophysiological targets to mitigate disease severity and 2) finding a more rational approach to repurposing existing drugs. It is increasingly recognized that acute viral disease severity is heavily driven by the immune response to the infection ("cytokine storm" or "cytokine release syndrome"). There exist numerous clinically available biologics that suppress various pro-inflammatory cytokines/mediators, but it is extremely difficult to identify clinically effective treatment regimens with these agents. We propose that this is a complex control problem that resists standard methods of developing treatment regimens and accomplishing this goal requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training successful game-playing artificial intelligences (AIs). This proof-of-concept study determines if simulated sepsis (e.g. infection-driven cytokine storm) can be controlled in the absence of effective antimicrobial agents by targeting cytokines for which FDA-approved biologics currently exist. Methods: We use a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. DRL training used a Deep Deterministic Policy Gradient (DDPG) approach with a clinically plausible control interval of 6 hours with manipulation of six cytokines for which there are existing drugs: Tumor Necrosis Factor (TNF), Interleukin-1 (IL-1), Interleukin-4 (IL-4), Interleukin-8 (IL-8), Interleukin-12 (IL-12) and Interferon-γ(IFNg). Results: DRL trained an AI policy that could improve outcomes from a baseline Recovered Rate of 61% to one with a Recovered Rate of 90% over ~21 days simulated time. This DRL policy was then tested on four different parameterizations not seen in training representing a range of host and microbe characteristics, demonstrating a range of improvement in Recovered Rate by +33% to +56. Discussion: The current proof-of-concept study demonstrates that significant disease severity mitigation can potentially be accomplished with existing anti-mediator drugs, but only through a multi-modal, adaptive treatment policy requiring implementation with an AI. While the actual clinical implementation of this approach is a projection for the future, the current goal of this work is to inspire the development of a research ecosystem that marries what is needed to improve the simulation models with the development of the sensing/assay technologies to collect the data needed to iteratively refine those models.

Navigating Robots in Dynamic Environment With Deep Reinforcement Learning

In the fight against COVID-19, many robots replace human employees in various tasks that involve a risk of infection. Among these tasks, the fundamental problem of navigating robots among crowds, named robot crowd navigation, remains open and challenging. Therefore, we propose HGAT-DRL, a heterogeneous GAT-based deep reinforcement learning algorithm. This algorithm encodes the constrained human-robot-coexisting environment in a heterogeneous graph consisting of four types of nodes. It also constructs an interactive agent-level representation for objects surrounding the robot, and incorporates the kinodynamic constraints from the non-holonomic motion model into the deep reinforcement learning (DRL) framework. Simulation results show that our proposed algorithm achieves a success rate of 92%, at least 6% higher than four baseline algorithms. Furthermore, the hardware experiment on a Fetch robot demonstrates our algorithm’s successful and convenient migration to real robots. IEEE

Deep Reinforcement Learning based Haptic Enhancement for Tele-Diagnosis

Tele-Diagnosis is beneficial for medical care in areas with inadequate resources, which helps control the spread of Covid-19 in the current pandemic. Most teleoperated diagnostics are dependent on humans, possibly leading to cognitive issue caused by distanced communication. In this paper, we propose a local haptic enhancement framework to facilitate the remote palpation. The deep deterministic policy gradient (DDPG) algorithm is exploited to compensate for signal transmission due to latency, allowing human to operate without the sense of delay. With the help of weighted recursive least squares (WRLS) method, the interactive force can be estimated on the patient's side despite the lack of force sensors. Fuzzy inference is used to diagnose and classify the extent of disease based on the estimated force and motion state on the remote side, thereby leveraging the remote sensory information to conduct autonomous reasoning. Finally, the final diagnosis is derived by performing minimum risk Bayesian decision based on local and remote inference results. Comparative simulation results have validated the superior performances of the proposed scheme. © 2022 IEEE.

Super-Resolution-Empowered Adaptive Medical Video Streaming in Telemedicine Systems

Due to influence of COVID-19, telemedicine is becoming more and more important. High-quality medical videos can provide a physician with a better visual experience and increase the accuracy of disease diagnosis, but this requires a dramatic increase in bandwidth compared to that required by regular videos. Existing adaptive video-streaming approaches cannot successfully provide high-resolution video-streaming services under poor or fluctuating network conditions with limited bandwidth. In this paper, we propose a super-resolution-empowered adaptive medical video streaming in telemedicine system (named SR-Telemedicine) to provide high quality of experience (QoE) videos for the physician while saving the network bandwidth. In SR-Telemedicine, very low-resolution video chunks are first transmitted from the patient to an edge computing node near the physician. Then, a video super-resolution (VSR) model is employed at the edge to reconstruct the low-resolution video chunks into high-resolution ones with an appropriate high-resolution level (such as 720p or 1080p). Furthermore, the neural network of VSR model is designed to be scalable and can be determined dynamically. Based on the time-varying computational capability of the edge computing node and the network condition, a double deep Q-Network (DDQN)-based algorithm is proposed to jointly select the optimal reconstructed high-resolution level and the scale of the VSR model. Finally, extensive experiments based on real-world traces are carried out, and the experimental results illustrate that the proposed SR-Telemedicine system can improve the QoE of medical videos by 17–79% compared to three baseline algorithms. © 2022 by the authors.

Predicting the COVID19 Trajectory with a Simulation Deep Reinforcement Learning Approach

COVID19 pandemic has severely impacted every government and citizen of the world. At the same time, it has emphasized the significance of governmental decisionmaking when facing a sudden outbreak. In this paper, we aim to aid governments in addressing the difficult problem of epidemic control planning by providing a disease trajectory and economic impacts. We study a SimulationDeep Reinforcement Learning (SiRL) methodology to predict the COVID19 pandemic's trajectory for the next three months considering different intervention strategies. Our experiments show that if no action is taken, and the current rate of vaccination is assumed, the daily cases could see an increase of 145%. Our Reinforcement Learning (RL) agent builds a compromise between the size of the infected population and the pandemicrelated economic costs. © 2022 IISE Annual Conference and Expo 2022. All rights reserved.

District-Coupled Epidemic Control via Deep Reinforcement Learning

The rapid spread of the Coronavirus (COVID-19) poses an unprecedented threat to the public health system and social economy, with approximately 500 million confirmed cases worldwide. Policymakers confront with high-stakes to make a decision on interventions to prevent the pandemic from further spreading, which is a dilemma between public health and a steady economy. However, the epidemic control problem has vast solution space and its internal dynamic is driven by population mobility, which makes it difficult for policymakers to find the optimal intervention strategy based on rules-of-thumb. In this paper, we propose a Deep Reinforcement Learning enabled Epidemic Control framework (DRL-EC) to make a decision on intervention to effectively alleviate the impacts of the epidemic outbreaks. Specifically, it is driven by reinforcement learning to learn the intervention policy autonomously for the policymaker, which can be adaptive to the various epidemic situation. Furthermore, District-Coupled Susceptible-Exposed-Infected-Recovered (DC-SEIR) model is hired to simulate the pandemic transmission between inter-district, which characterize the spatial and temporal nature of infectious disease transmission simultaneously. Extensive experimental results on a real-world dataset, the Omicron local outbreaks in China, demonstrate the superiority of the DRL-EC compared with the strategy based on rules-of-thumb. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Multi-robot Sanitization of Railway Stations Based on Deep Q-Learning

Sanitizing railway stations is a relevant issue especially due to the recent evolution of the Covid-19 pandemic. In this work, we propose a multi-robot approach to sanitize railway stations based on a distributed Deep Q-Learning technique. The framework relies on anonymous information from existing WiFi networks to localize passengers inside the station and to develop a map of possible risky areas to be sanitized. Starting from this map, a swarm of cleaning robots, each one endowed with a robot-specific convolutional neural network, learns how to on-line cooperate inside the station in order to maximize the sanitized area depending on the presence of the passengers. © 2021 Copyright for this paper by its authors.

Permissioned Blockchain and Deep Reinforcement Learning Enabled Security and Energy Efficient Healthcare Internet of Things

Recently, the Healthcare Internet of Things (H-IoT) has been widely applied to alleviate the global challenge of the coronavirus disease 2019 (COVID-19) pandemic. However, security and limited energy capacity issues remain the two main factors that prevent the large-scale application of the H-IoT. Therefore, a permissioned blockchain and deep reinforcement learning (DRL)-empowered H-IoT system is presented in this research to address these two issues. The proposed H-IoT system can provide real-time security and energy-efficient healthcare services to control the propagation of the COVID-19 pandemic. To address the security issue, a permissioned blockchain method is adopted to guarantee the security of the proposed H-IoT system. As for handling the limited energy constraint, we employ the mobile edge computing (MEC) method to offload the computing tasks to alleviate the computational burden and energy consumption of the proposed H-IoT system. We also adopt an energy harvesting method to improve performance. In addition, a DRL method is employed to jointly optimize both the security and energy efficiency performance of the proposed system. The simulation results demonstrate that the proposed solution can balance the requirements of security and energy efficiency issues and hence can better respond to the COVID-19 pandemic.

Deep Reinforcement Learning Interdependent Healthcare Critical Infrastructure Simulation model for Dynamically Varying COVID-19 scenario – A case study of a Metro City

Inability to respond to the growing trend of COVID -19 cases and the study and analysis of Healthcare Critical Infrastructure interdependencies during COVID-19 pandemic scenario is relatively new. One of the most frequently identified shortfalls in knowledge related to enhancing Healthcare Critical Infrastructure (HCI) preparedness during the COVID-19 pandemic scenario is the inability to forecast the growth trend of COVID-19 cases in a geographic area and incomplete understanding of interdependencies between Critical infrastructures related to HCI. As the number of cases surges at a healthcare facility, the facility, and its interdependent CI services should be prepared to handle the susceptible stress. The goal of the paper is to be able to predict the growth trend of COVID-19 cases using Spatiotemporal Long Short-Term Memory (ST-LSTM) for a geographic area. Based on the predicted growth trend of the COVID-19 cases a Multi-Agent Deep Reinforcement Learning (MADRL) simulation model will provide an accurate representation of healthcare critical infrastructure characteristics, operations, and interdependencies services. The Real-time information simulation would help frontline workers, government agencies, and disaster and emergency response personnel to respond to the question, ‘what if something else happens during the COVID-19 Pandemic? © 2021 IEEE

Deep Reinforcement Learning-based Vaccine Distribution Strategies

The rapid development of artificial intelligence techniques is significantly promoting the resolution of various important decision-making issues such as material distribution, generation line optimization scheduling, and path planning. Currently, SARS-CoV-2 is raging over the world, and it is valuable to propose a vaccine distribution strategy to utilize limited vaccine resources rationally. In this paper, we aim to propose an optimal vaccine distribution strategy based on deep reinforcement learning(DRL) approaches. An End-to-End vaccine distribution model is proposed by combining the Deep Reinforcement Learning model and LinUCB algorithm to get an optimistic strategy of allocation. Experiment results demonstrated that vaccine distribution strategies based on this model show a strong capacity to control the epidemic and ensure stable government revenue compared with baseline strategies. © 2021 IEEE.

A deep reinforcement learning based decision-making approach for avoiding crowd situation within the case of Covid'19 pandemic.

Individuals' flow's fluidifcation in the same way as the thinning of the population's concentration remains among major concerns within the context of the pandemic crisis situations. The recent COVID-19 pandemic crisis is a typical example of the aforementioned where on despite of the containment phases that radically isolate the population but are not applicable persistently, people have to adapt their behavior to new daily-life situations tempering Individuals' stream, avoiding tides, and watering down population's concentration. Crowd evacuation is one of the well-known research domains that can play a pertinent role to face the challenge of the COVID-19 pandemic. In fact, considering the population's concentration thinning within the slant of the "crowd evacuation" paradigm allows managing the flow of the population, and consequently, decreasing the probable number of infected cases. In other words, crowd evacuation modeling and simulation with the aim of better-exploiting individuals' flow allow the study and analysis of different possible outcomes for designing population's concentration thinning strategies. In this article, a new decision-making approach is proposed in order to cope with the aforesaid challenges, which relies on an independent Deep Q Network with an improved SIR model (IDQN-I-SIR). The machine-learning component (i.e., IDQN) is in charge of the agent's movements control and I-SIR (improved "susceptible-infected-recovered" individuals) model is responsible to control the virus spread. We demonstrate the effectiveness of IDQN-I-SIR through a case-study of individuals' flow's management with infected cases' avoidance in an emergency department (often overcrowded in context of a pandemic crisis).

Flow Control in Wings and Discovery of Novel Approaches via Deep Reinforcement Learning

In this review, we summarize existing trends of flow control used to improve the aerodynamic efficiency of wings. We first discuss active methods to control turbulence, starting with flat-plate geometries and building towards the more complicated flow around wings. Then, we discuss active approaches to control separation, a crucial aspect towards achieving a high aerodynamic efficiency. Furthermore, we highlight methods relying on turbulence simulation, and discuss various levels of modeling. Finally, we thoroughly revise data-driven methods and their application to flow control, and focus on deep reinforcement learning (DRL). We conclude that this methodology has the potential to discover novel control strategies in complex turbulent flows of aerodynamic relevance.